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ABSTRACT KEYWORDS 
Community colleges are a large sector of postsecondary education. community college 

In 2016-2017, the United States had nearly 1,000 public 2-year post- randomized controlled trial 
secondary institutions (community colleges), serving almost nine mil- developmental math 

lion students, representing 39% of all undergraduates. The majority — ™ularized 4 

of entering community college students require developmental (or ee 


remedial) math. Success rates in the developmental math course 
sequence and college more broadly are discouragingly low. 
Policymakers, practitioners, and researchers alike are eagerly search- 
ing for reforms to improve success rates, but there is a dearth of 
causal evidence on the effectiveness of most proposed reforms. We 
sought to answer the following question: what effect does a modu- 
larized, computer-assisted, self-paced approach to developmental 
math (compared with a more “traditional” direct-instruction course 
alternative) have on students’ likelihood of completing the develop- 
mental math course sequence? Findings from a randomized 
controlled trial (n =1,403) are presented. The program was well 
implemented; however, we did not find evidence that this approach 
was superior to the “traditional” math class. Although these results 
are disappointing, they are important because modularization and 
self-paced computer-assisted instruction are popular reforms. 


Introduction 


In the United States, community colleges play a vital role in advancing the nation’s agenda 
to increase college degree attainment and technical skills training, as they serve a signifi- 
cant proportion of postsecondary education students. In 2016-2017, nine million students 
attended public 2-year colleges (community colleges), representing 39% of all undergradu- 
ates. The open admissions and relatively low cost of community colleges (compared with 
4-year colleges and universities) have contributed to unprecedented access to postsecondary 
education. However, rates of successful degree completion leave much room for improve- 
ment. Nationwide, among first-time, full-time community college students in the 2013 
cohort, only 25% graduated within 3 years (Ginder, Kelly-Reid, & Mann, 2017). 

One of the greatest challenges facing community colleges is that most entering stu- 
dents (approximately 59%) are deemed academically underprepared for college-level 
math and are referred to developmental math courses (Chen, 2016). These courses are 
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intended to prepare students for college-level work; however, most students who place 
into remedial math courses never complete them, move on to introductory college-level 
math courses, or earn a degree (Attewell, Lavin, Domina, & Levey, 2006; Bailey, Jenkins, 
& Leinbach, 2005; Bailey, Jeong, & Cho, 2010). 

While the low success rates of students who are referred to developmental educa- 
tion are well documented (Bailey et al., 2010), there is a dearth of causal evidence 
on the effectiveness of strategies to improve outcomes for these students. In fact, the 
U.S. Department of Education’s What Works Clearinghouse (WWC) recently pub- 
lished a practice guide of strategies to help postsecondary students in developmental 
education and after reviewing 25,697 studies, only 10 met the WWC evidence stand- 
ards with or without reservations. Moreover, of the six practices recommended in 
the guide, only three had any evaluations meeting the WWC evidence standards sup- 
porting them, indicating that causal evidence on strategies to improve developmental 
education is desperately needed by postsecondary administrators and policy makers 
(Bailey et al., 2016). 

The present study contributes to this literature via a rigorous evaluation of a widespread 
type of developmental education reform—the division of remedial math courses into dis- 
crete, single-unit modules, in which the content is delivered via self-paced, computer- 
assisted instruction. The popularity of this type of reform extends from statewide policies 
to independent creation of homegrown programs at individual colleges. A 2016 survey of a 
nationally representative sample of 911 2- and 4-year colleges in the United States found 
that 40% of institutions offered self-paced approaches to developmental math education 
and 32% of colleges used computer-based learning to support underprepared students 
(Zachry Rutschow & Mayer, 2018, personal communication). In the past few years, policy 
makers in Virginia and North Carolina independently standardized the developmental edu- 
cation curricula across all the colleges in their respective states and modularized their devel- 
opmental math courses (Kalamkarian, Raufman, & Edgecombe, 2015). 

At Tarrant County College (TCC) in Texas, math department faculty created 
ModMath, a modularized, computer-assisted, self-paced developmental math course 
sequence, which is the subject of the present study. In this article, we present findings 
from a 1,400-person randomized controlled trial of ModMath. The sections that follow 
provide context for the study, describe the program model and theory of change, and 
detail the study design and results. We conclude with a discussion on the implications 
of the findings. 


Background and Study Context 


Developmental Math at TCC 


When the present study began, the developmental math sequence at TCC consisted of 
two courses: Developmental Math 1 and Intermediate Algebra. Students were referred 
to one of the two developmental math classes or into college-level math using the Texas 
Success Initiative (TSI) Assessment.’ The courses, which all followed the same 


"Students may be exempt from taking the TSI Assessment if they obtain a certain score on their SAT, ACT, or Texas statewide 
high school test; have successfully completed a college-level math course; or are active or veteran members of the military. 
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Table 1. Features of developmental math course offerings at Tarrant County College. 


Course type Sequence structure Instructional delivery 
ModMath e Six 5-week modules e Computer-assisted instruction 
e Each mod is 1 credit e —_Self-paced 
e Support from instructor and aide 
e Acceleration possible 
Lecture-based e Two 16-week courses e —_Instructor-led lectures 
e Each course is 3 credits e Instructor sets pace 
Computer-assisted lecture e Two 16-week courses e —_Instructor-led lectures and 
e Each course is 3 credits computer assisted instruction 


(varies by instructor) 
e Instructor sets pace 


Emporium e Two 16-week courses e Computer-assisted instruction 
e Each course is 3 credits e —_Self-paced 
e Acceleration possible 


curriculum, were offered in a variety of course formats, including lecture-based courses, 
computer-assisted lectures, Emporium (described later), and ModMath. 

The focus of this article is the effectiveness of TCC’s ModMath program. However, as 
Holland (1986) noted, “The effect of a cause is always relative to another cause” (p. 
946). In other words, there is no singular effect of TCC’s ModMath program—it 
depends what it is being compared with. In the present study, we examine the effective- 
ness of TCC’s ModMath relative to the alternative developmental math course offerings 
at TCC. Table 1 provides a brief overview of ModMath and these alternative offerings, 
which is essential for understanding the service contrast and thus the estimated effects 
on academic outcomes. While the control group’s experiences reflect some participation 
in all three alternatives, most control group students enrolled in lecture-based courses. 
In the Program Implementation section, we provide greater detail on participation in 
these alternative offerings and students’ experiences in these courses. 


ModMath Model, Theory of Change, and Prior Research 
Program Model 


ModMath is a developmental math reform that changes the structure of the develop- 
mental math sequence and the instructional delivery of the curriculum, but not the 
course content itself. ModMath’s course structure and instruction encompass four key 
components: (1) modularized courses, (2) computer-assisted instruction, (3) a diagnostic 
assessment, and (4) on-demand, personalized assistance. Figure 1 depicts the ModMath 
theory of change or logic model. The first two columns describe each component of the 
program and associated practices. The last two columns list intended student outcomes, 
and the middle column explains the mechanisms through which each set of practices 
was hypothesized to improve outcomes for ModMath students. 

Modular Courses: The core component of ModMath is a structural change that 
divides each of the two semester-long developmental math courses into three 5-week 
one-credit modules or “mods” (mods 1-6). All six modules are offered in any given 
ModMath course section. Students typically enroll in three modules—the equivalent of 
one traditional developmental math course—each semester. 


486 M. J. WEISS AND C. HEADLAM 


Component Key Practices and Features Mechanisms Student Outcomes 
7 i Fine-tuned for module placement o Accurate placement resulting in close = 
oe Aligned with course content alignment of content with students’ prior Short-Term Long-Term 
(MyMathTest) math knowledge Academic Progress Academic Progress 
Each semester-long course divided o Students earn one credit for each mod 
into three 5-week modules passed, resulting in sense of progress 
Each module worth 1 co Students who fail or stop attending a mod 
Modularized developmental math credit can repeat in the next 5-week session 
Courses Modules align with the standard rather than wait until next semester to 
curriculum repeat whole course, resulting in Enrollment ©  College-level credits 
increased persistence Proportion of the earned 
100 percent of class time in o Allows for self-paced learning developmental math | o Total credits earned 
computer lab co Allows for completing up to 6 modules in Sequence: completed |o Persistence ead 
Computer- Various content delivery methods a single semester amen te es 9 pate ee eae 
Based for instruction (video, presentation, o Students move on only after ea pte 4 lon 7 statis ied 
Instruction textbook) demonstrating mastery of material . pean Tlie oe 2 Sais ra rae ioe 
(MyMathLab) Mastery learning o Variety of content delivery methods and ee ue SOmMer ian year colleges 
= Extra support available via frequent assessments facilitates mastery mall 
software 
Each class staffed with an o Increases the amount of one-on-one 
Personalized. instructor and aide instructor-student interactions, allowing 
On-Demand : Instructor and aide circulate during more academic and emotional support 
‘Aasistance class, providing one-on-one 
assistance 


Figure 1. A logic model for ModMath: Components, practices, mechanisms, and outcomes. 


Computer-Assisted Instruction: ModMath’s course content is delivered via Pearson’s 
instructional software program called MyMathLab. Students meet in computer class- 
rooms at regularly scheduled times and work independently through the course material 
using instructional videos, PowerPoint slides, or an online or hard copy version of the 
textbook. Because instruction is self-paced, students may accelerate and complete more 
than three modules in each semester. 

Diagnostic Assessment: To place students into one of the six computer-assisted mod- 
ules, ModMath supplements the college’s standard math placement test—the TSI 
Assessment—with an additional placement exam, Pearson’s MyMathTest.7 MyMathTest 
is intended to be more fine-tuned than TCC’s standard placement exam, identifying 
which module matches students’ demonstrated math knowledge and skill deficits. 
Students are referred to a specific module depending on their skill needs as determined 
by MyMathTest results and must complete the remaining set of modules in order to 
complete developmental math. 

On-Demand, Personalized Assistance: ModMath is designed so instructors serve as 
facilitators who provide individualized support to students, as opposed to whole-group 
instruction. Each class, which typically enrolls 24 students, is staffed with an instructor 
and an instructional aide. Compared with traditional lecture-based courses, the highly 
personalized structure of ModMath provides more opportunity for instructional staff to 
provide one-on-one academic and emotional support to students. 


Theory of Change 


ModMath is theorized to improve student outcomes by addressing common challenges 
to developmental math instruction. A major pedagogical challenge in developmental 


The exam, which is developed using Pearson’s MyMathTest platform, allows faculty to create a customized placement 
exam by selecting questions from the software's test bank of problems that align with TCC’s developmental 
math curriculum. 
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math is that the courses typically serve heterogenous students with a wide range of aca- 
demic abilities, learning styles, and personal needs. Unless an instructor can accommo- 
date these differences through differentiated instruction and other methods, such as 
supplemental academic support, students may fall behind, disengage, or fail (Tomlinson 
& Kalbfleisch, 1998). Several ModMath components allow for greater differentiation 
than is typically feasible in traditional lecture-based classrooms. 

First, the modular structure, coupled with the fine-tuned diagnostic exam, is intended 
to allow for more precise placement than the standard assessment. This should allow 
students to enter a modular sequence at a level that is closely aligned with their prior 
knowledge, so they focus only on topics in which they need remediation (Bickerstaff, 
Fay, & Trimble, 2016; Bracco, Austin, Bugler, & Finkelstein, 2015). Second, ModMath 
tailors instruction to each student by combining computer-based learning via the 
instructional software package with on-demand, personalized assistance from the 
instructor and aide. This “hybrid” learning environment has been linked to improved 
student outcomes (Chekour, 2017; Means, Toyama, Murphy, Bakia, & Jones, 2009); 
although the evidence is mixed (Xu & Jaggars, 2011). Instructional software is thought 
to accommodate students with various learning styles and abilities by providing diverse 
instructional materials and allowing for self-paced instruction. Both features are 
intended to increase student engagement and performance (Goldschmid & Goldschmid, 
1973; Subban, 2006). Furthermore, self-paced instruction should allow some students to 
accelerate and complete more than the standard three modules per semester. 

In general, instructional software, such as MyMathLab, is theorized to facilitate opti- 
mal learning by placing students in Vygotsky’s (1980) zone of proximal development— 
the developmental level where instructional content is just beyond students’ current 
knowledge—a level where students are capable of learning new material with assistance 
from a more knowledgeable tutor. In ModMath, this tutoring can come from the soft- 
ware, which includes several help features to guide students through new math prob- 
lems, or from the instructor, who can fill in conceptual gaps or provide alternative 
methods of problem solving to those provided by the software. Computer-assisted 
instruction is also theorized to facilitate mastery learning because it allows students to 
take frequent assessments, receive formative feedback on their learning, and progress 
only when they have mastered the material. This process is thought to improve student 
performance in developmental and in more advanced math courses (Bishop, 2010; 
Epper & Baker, 2009; Twigg, 2005). 

Another common challenge to developmental math instruction is that many students 
have a history of underperformance in math, suffer from math anxiety, or lack confi- 
dence in their mathematical abilities (Dwinell & Higbee, 1989; Taylor, 2006). By staffing 
each course with an instructor and aide who provide individualized assistance, 
ModMath increases instructor-student interactions and allows opportunities for instruc- 
tional staff to provide not only academic but also emotional support to students. This 
support may build student confidence and improve academic progress in math, as 
greater connections to staff can increase student feelings of belonging and facilitate per- 
sistence (Tinto, 1999). 

Finally, another challenge to developmental math instruction, especially in commu- 
nity colleges, is that many students stop attending classes or drop out of college. In 
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traditional courses, students who drop out mid-semester do not receive any credit for 
already completed work and must restart the course from the beginning. The modular- 
ized structure allows students to leave and return, without losing as much ground. 
Furthermore, students using ModMath earn credits incrementally, which may facilitate a 
sense of accomplishment as they experience “small wins” (Weick, 1984) toward the goal 
of completing the entire developmental math sequence. 


Effectiveness of Modularized, Computer-Assisted Developmental Math Programs 


There is limited rigorous evidence on the efficacy of modularized, computer- 
assisted developmental math courses, like ModMath, and findings from existing non-ex- 
perimental studies are mixed. For example, Squires, Faulkner, and Hite (2009) compare 
developmental (and college-level) course completion rates pre- and post-computer-- 
assisted modularization and find increases in pass rates. On the other hand, 
Kalamkarian et al. (2015) studied North Carolina’s and Virginia’s modularized develop- 
mental math programs and, based largely on descriptive outcome data, concluded that 
pacing and attrition were major issues in self-paced, computer-assisted modules, as a 
significant portion of students did not complete the expected number of modules for 
the given semester. In another study, Ariovich and Walker (2014) compared the pass 
rates of students in modularized classes to pass rates of students in traditional courses 
(without controlling for any student background characteristics). They found that stu- 
dents who opted to take modularized, computer-assisted developmental math performed 
worse than students who opted to take traditional developmental math. 

The mixed results of modularized, computer-assisted, self-paced developmental math 
courses may relate to variations due to program implementation and institutional con- 
text. Fay (2017) concluded that high school students in Tennessee experienced higher 
pass rates than community college students using the same computer-assisted modular- 
ized developmental math program because the high schools had more structured class- 
room environments, such as rigorous attendance policies and frequent class meetings, 
than the community colleges, which were characterized by more flexibility and student 
autonomy. Bickerstaff et al. (2016) found that stand-alone modules (in which all stu- 
dents are in the same module) allow for instructional flexibility (as they can be taught 
via lecture or instructional software) but create additional exit points at which students 
may fail to reenroll in the next module. On the other hand, computer-assisted courses 
that offer several modules in one course section reduce the number of exit points but 
may slow student progression. 

At least one quasi-experimental study has been conducted. Following a statewide 
redesign of remedial education in Tennessee, some community colleges divided their 
developmental math sequence into computer-assisted modules. Boatman (2012) eval- 
uated two programs of this type and found similarly mixed results: The computer- 
assisted modularized courses resulted in positive effects at one institution, but no effects 
at the other community college.’ 


3This study uses a complicated design that combines regression discontinuity, instrumental variables, and difference-in- 
differences. 
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The inconclusive literature on modularized, computer-assisted courses highlights the 
need for randomized field trials to test the efficacy of the approach. The present study 
aims to begin to fill this gap. 


Evaluation 
The Study College: TCC, Texas 


The study took place at the TCC District. With six campuses and an annual enrollment 
of about 50,000 students, TCC is one of the largest community college systems in 
Texas. This study took place at the Northeast Campus, located in Hurst, a suburb 
between Fort Worth and Dallas, Texas. The campus offers a range of associate degree 
programs that prepare students to enter professional careers or transfer to 4-year 
institutions. 


Research Questions 


The primary goal of the evaluation is to answer the following question: 


e What effect does the opportunity to enroll in ModMath (compared with the 
opportunity to enroll in the college’s “traditional” math courses) have on stu- 
dents’ likelihood of completing the developmental math course sequence?* 


We were also interested in any positive spillover effects or negative side effects on 
progress outside of math, which may be caused by ModMath. 

In addition to this overarching goal, we sought to understand several questions 
related to the implementation of ModMath and the “traditional” math courses with 
which ModMath is compared. They include the following: 


e To what degree were ModMath services and activities implemented as 
planned (i.e., to what degree is their fidelity to the program model)? 

e To what degree were the services experienced by program group students dif- 
ferent from those experienced by control group students (i.e., to what degree 
was there a contrast between the program and control conditions)? 


Research Design 


We used a random assignment research design to estimate the effect of the ModMath 
program compared with a “business-as-usual” control condition at the college, generally 
a more traditional, lecture-based course. Included in the evaluation were eligible stu- 
dents (those in need of developmental mathematics, based on placement test scores) 


‘This question and the implied primary outcome of interest were specified in an internal (i.e., not published) analysis 
plan written prior to conducting analyses. The main outcome (confirmatory) of interest was having successfully 
completed the developmental math course sequence (for more on what is meant by “confirmatory,” please see 
Schochet, 2008). 
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Table 2. Analytic sample of students by cohort and experimental group. 


Program group Control group 
Cohort (n= 826) (n= 577) 
Spring 2014 86 79 
Fall 2014 285 201 
Spring 2015 133 85 
Fall 2015 322 212 


Note. N= 1,403. Calculations made using MDRC’s random assignment system. 


who (a) were willing to participate in the ModMath program, (b) filled out a baseline 
survey, and (c) signed an informed consent.’ After completing the baseline survey and 
informed consent, students were randomly assigned, through a computer algorithm con- 
trolled by the research team, to either the program or control group. Program group 
members had the opportunity to participate in ModMath. Control group members had 
the opportunity to participate in any of the college’s other developmental math course 
offerings and any support services, just not ModMath. Four cohorts of students were 
randomly assigned, one prior to the spring and fall semesters in 2014 and 2015. In total, 
1,408 students were randomly assigned (828 to the program group, 580 to the control 
group). Five students are not included in any analyses (two in the program group, three 
in the control group) because they withdrew from the study or their consent form was 
not recovered, leaving an analytical sample of 1,403 students.° Table 2 provides the size 
of the analytic sample by cohort and experimental group. Most students (73%) entered 
the study in the two fall semesters. Overall, 59% of the sample was assigned to the pro- 
gram group. The proportion was smaller for the Spring 2014 cohort because the original 
research design involved a 1:1 random assignment ratio. This was modified to a 3:2 
ratio for the three later cohorts to increase the number of students interested in 
ModMath who were offered the program. 


Student Characteristics and Baseline Balance Tests 


Upon joining the study, students completed a baseline survey covering information about 
their demographic characteristics, family and educational backgrounds, and experiences 
with math. These data are used to describe the sample and compare the pretreatment 
measured characteristics of program and control group sample members. Overall the 
sample is diverse and demographically similar to community college students nationally, 
and the program and control groups were very similar at the outset of the study. 

Table 3 shows that, as is the case in the undergraduate population nationally, most 
students in the study sample are female. The analytic sample is racially diverse: 46% 
White, 29% Hispanic, and 19% Black. This is similar to the racial demographics at pub- 
lic institutions nationally, which are 50% White, 25% Hispanic, and 14% Black (Snyder, 
de Brey, & Dillow, 2018). At the start of the study, 60% of students were younger than 
25 years old; 22% were between 25 and 34years old; and about 18% were older than 


5For more details on the recruitment process, see Gardenhire, Diamond, Headlam, and Weiss (2016). 
‘The overall attrition rate is 0.36% and the rate of differential attrition (the difference between program group attrition 
and control group attrition) is 0.28 percentage points. 
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Table 3. Characteristics of students in the study. 


Program group Control group Standard 

Outcome (%) (n = 823) (n= 575) Difference error 
Female 63.5 65.0 —1.5 2.6 
Age 

18 and younger 22.1 21.6 0.5 2.2 

19-24 38.2 38.8 —0.6 2.6 

25-34 22.0 21.4 0.6 2.2 

35-44 10.0 11.1 —1.2 1.7 

45 and older 78 7.1 0.6 1.4 
Race/ethnicity* 

Hispanic 28.7 27.3 1.4 2.5 

White 46.2 45.4 0.7 2.8 

Black 19.2 20.1 —0.9 2.2 

Other 5.9 72 -13 14 
Completed 12th grade 86.8 88.1 —13 18 
First person in family to attend college 32.9 34.8 —1.9 2.6 
Planned enrollment this semester 

Less than part time (fewer than 6 credits) 17.2 18.5 —13 24 

Part time (6-12 credits) 39.0 38.9 0.1 2.7 

Full time (12 credits or more) 43.8 42.6 12 27 
Planning to work this semester 

No 19.2 18.8 0.4 2.1 

Yes, part time (less than 30 hours a week) 29.9 323 =25 2.5 

Yes, full time (30 hours a week or more) 443 42.4 1.9 27 

Missing 6.6 6.4 0.1 1.3 
Failed a math class in the past 52.9 54.3 —14 2.7 

Missing 6.3 7.3 —1.0 1.4 
TSI math placement 

College-ready or exempt” 7.2 6.3 0.9 14 

Placed one level below college-ready 9.2 8.7 0.5 1.6 

Placed more than one level below college-ready 50.5 54.1 —3.5 27. 

Math placement information is 

unknown or missing‘ 33.0 31.0 a 2.5 


Note. N= 1,398. Rounding may cause slight discrepancies in sums and differences. Missing shows the percentage of sur- 
vey respondents who did not answer the question. Missing values are reported only for items with more than 5% miss- 
ing. Calculations were made using data from the baseline survey of Tarrant County College (TCC) students and TCC 
placement test data. Five students never filled out baseline information surveys and are excluded from this table. 

TSI = Texas Success Initiative. 

*Respondents who said they were Hispanic and chose a race are included only in the Hispanic category. Respondents 
who said they were not Hispanic and chose more than one race are included in the Other category. The Other cat- 
egory also includes respondents who chose Asian, American Indian, or Pacific Islander. 

includes students who were found to be ready for college, who received waivers from testing requirements, who were 
exempt from testing requirements, or who had previously completed testing requirements. 

‘Includes students who were not included in the TCC placement test data. 

***n = 01; **p = 05; *p = .10. 


35 years old. This age breakdown mirrors national statistics among part-time students, 
whereas full-time students tend to be younger (NCES, 2018a). Many students in the 
study had characteristics associated with a low likelihood of academic success (Engle, 
2007). For instance, a third of the students in the study are the first in their families to 
attend college.’ Around 43% of the analytic sample reported that they planned to enroll 
in school full-time during the first study semester.* Nearly three-quarters of sample 
members said that they planned to work during the upcoming semester, and about 44% 
planned to work full time.’ In addition, more than half of students reported that they 


7Nationally, 24% of college students are first-generation college students (Redford & Hoyer, 2017). 
8similarly, 37% of students at 2-year public institutions enrolled full time in 2015 (NCES, 2018a). 


°similarly, at 2-year institutions nationally, 45% of part-time undergraduate students worked 35 hours or more per week 
in 2015; 46% of full-time and 76% of part-time students held employment (NCES, 2018b). 
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had failed a math class in the past. Also shown in Table 3, the pretreatment characteris- 
tics of program and control group students are very similar.'° 


Instructor Characteristics 


ModMath instructors were selected in one of two ways: Full-time instructors could vol- 
unteer to teach ModMath, while adjunct instructors were assigned to teach ModMath 
by the math department chairs. Developmental math instructors were surveyed about 
their demographic characteristics, educational and professional backgrounds, and teach- 
ing experiences to gauge whether the instructors teaching ModMath might differ from 
other instructors, particularly in ways that could affect students’ academic outcomes. 
For example, if ModMath classes were taught by more experienced instructors, and if 
instructor experience is associated with effectiveness, then program group students 
might be more likely than control group students to pass developmental math not 
because the ModMath program itself is more effective, but because its instructors had 
more experience (see Weiss, 2010 for details on the implications of instructor selection 
in individually randomized trials). 

Table 4 shows that the two groups of instructors were similar in many regards. Most 
instructors in both groups were White, with an average age of around 48 or 49. 
Instructors in both groups reported having taught for an average of 17 years and having 
taught developmental math specifically for 10years. Both groups were also roughly 
equally likely to have received different types of professional development within the 
past 2 years. 

In other ways, however, the groups were different. For example, ModMath instructors 
were more likely to be female (64%) than other instructors (53%). ModMath instructors 
were also somewhat more involved in the math department than other instructors: 
They were more likely to have served as math department administrators (17% vs. 11% 
of other instructors) or to have served on a committee on math curriculum, evaluation, 
or assessment (32% vs. 26%). These differences should be kept in mind when consider- 
ing the findings. 


Data Sources 


Data for this mixed methods evaluation came from qualitative and quantitative sources. 
These data sources are briefly described below; for more details, see Gardenhire, 
Diamond, Headlam, and Weiss (2016). 

Qualitative data were collected from focus groups with five groups of students (rang- 
ing in size from 4 to 20 participants), one group of ModMath instructors and one group 
of traditional developmental math instructors (17 participants total), and one group of 
academic advisers (seven advisers). Interviews with 12 TCC staff members involved in 
the ModMath program were conducted. Focus groups and interviews covered typical 
student and instructor assessment and instructional activities; reflection on course struc- 
ture, pacing, and difficulty; student progression through each course type; and 


1cee Appendix A.1 in Gardenhire et al. (2016) for a comprehensive list of data reported by students on the baseline 
survey, as well as a comparison of program and control group students on these measured characteristics. 
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Table 4. Characteristics of developmental math instructors. 


Sample ModMath Other Standard 
Response size instructors instructors _ Difference error 
Male (%) 4 36 47 —11 15.7 
Age (years) 28 48 49 0 6.6 
Missing (%) 42 22 47 —26* 14.4 
Race/ethnicity* (%) 
Hispanic 42 0 5 —5 48 
White 42 83 74 9 13.0 
Asian 42 0 5 —5 48 
Black 42 4 5 -1 6.8 
American/Indian 42 4 0 4 48 
Missing 42 9 11 —2 9.3 
Years of teaching experience 
Developmental math 41 10 10 0 3.5 
All subjects 42 17 17 -1 4.0 
Adjunct or full faculty member (%) 
Adjunct 42 61 68 -8 15.2 
Full faculty member 42 35 32 3 15.0 
Other 42 4 0 4 48 
Highest degree earned (%) 
Bachelor's 42 9 11 —2 9.3 
Master's 42 83 79 4 12.5 
Doctorate 42 9 11 —2 9.3 
Current or former math department administrator (%) 42 17 11 i bla el 
For degrees earned, majors or concentrations” (%) 
Mathematics/statistics/economics 42 87 84 3 11.1 
Education (focused on math education) 42 35 42 =] 15.4 
Education (other focus) 42 17 5 12 10.1 
Engineering/computer science/science 42 48 32 16 15.4 
Other 42 17 5 12 10.1 
Participated in a committee for math curriculum, 
evaluation, or assessment (%) 41 32 26 6 14.6 
Participated in a campus-level committee (%) 42 26 32 —5 14.3 
Within the past 2 years, received 
professional development in? (%) 
Math content knowledge 41 68 63 5 15.2 
Computer-assisted instruction Al 59 47 12 15.9 
Teaching methods/pedagogy 39 81 78 3 13.3 
Average instructor agreement regarding 
students (1 = very true, 4 = not very true) 
They can understand the material 42 15 1.6 —0.1 0.2 
They will succeed in college algebra 40 2.0 2.2 —0.1 0.2 
They don’t study enough 42 1.8 17 0.1 0.2 


Note. Total survey respondents N= 42. ModMath respondents N= 23. Other respondents N= 19. Rounding may cause 
slight discrepancies in sums and differences. “Missing” shows the percentage of survey respondents who did not answer 
the question. Missing values are only reported for items with more than 5% missing. Calculations made using data from 
survey of Tarrant County College developmental math instructors. 

*Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 

>Distributions may not sum to 100% because categories are not mutually exclusive. 

***n = 01; **p = 05; *p = .10. 


discussion on institutional and course-specific developmental math education policies. 
Finally, the research team informally observed ModMath classrooms 10 times and con- 
trol group classrooms 5 times. Observers recorded data on the classroom environment, 
student and staff activities throughout the class period, and student engagement during 
the instructional period. The qualitative data were used primarily to understand the pro- 
gram implementation and similarities and differences in the experiences of program and 
control group members. 
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Quantitative data include a baseline survey taken just prior to random assignment; 
TSI placement test records, which were used to determine developmental math course 
placement for most control group students; MyMathTest placement scores, which were 
used to determine module level for most program group students; student transcript 
records, which we use to measure academic outcomes; a student survey conducted dur- 
ing program and control group students’ first semester after random assignment, cover- 
ing topics such as student engagement in their math courses, typical in-class activities 
by the student and instructor, and assessment of the difficulty and pacing of their math 
courses; and an instructor survey given to all math instructors who taught at least one 
developmental math class between Spring 2014 and Fall 2015. The student survey had 
an overall response rate of 72% (1,012/1,403). Program group members’ response rate 
was 75% (620/826) and control group members response rate was 68% (392/577). The 
instructor survey had an overall response rate of 82% (42/51)."" 


Fidelity of Implementation 


Fidelity to the ModMath model was assessed based on adherence to the four core compo- 
nents of the program model—modular courses, diagnostic assessment, computer-assisted 
instruction, and on-demand, personalized assistance. These components were identified 
from prior research on the program and from explicit discussions with the ModMath 
program designers about the program model and theory of change. For each core compo- 
nent, strong fidelity was determined to be the availability of the core component for most 
students, while weak fidelity was determined to be the absence of the core component for 
most students. Process mechanisms or the extent to which the program components and 
practices led to a modification in behaviors for most students were also examined. 

Course transcript data were used to assess whether ModMath courses were offered as 
discrete one-credit modules and whether most program participants took ModMath. 
Transcript data were also used to understand how students progressed through the 
modules and earned credits. Interviews and focus groups with staff and students con- 
tributed to understanding how the modular structure impacted student behavior and 
sense of achievement. Staff interviews and course documents were used to assess cur- 
riculum alignment—or the extent to which ModMath covered the same content as the 
lecture-based courses, as planned. Placement test data were used to examine adherence 
to the fine-tuned assessment requirement, measured by the proportion of students who 
took the exam prior to their first ModMath course, as prescribed by the program model. 
As a process measure, students’ placement using the fine-tuned exam was compared to 
their placement with the standard exam to determine whether the ModMath diagnostic 
altered placement for most students. 

For computer-assisted instruction and on-demand personalized assistance, classroom 
observation data were used to understand the classroom practice, and survey data were 
used to assess whether most students spent the majority of their math class time working 
independently on the computer software with assistance from the instructor as requested 
or needed. As process measures, student focus group and course transcript data were used 


it is difficult to determine response rates for ModMath instructors and other instructors because some instructors 
taught both types of classes, and we are not certain what types of classes nonrespondents taught. 
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to measure student pacing and acceleration, and focus group data were used to understand 
how on-demand, personalized assistance impacted student engagement and support. 

Qualitative data were collected at two points—once in each of the first two semesters 
following random assignment. Following each field visit, transcripts of interviews and 
focus groups were coded using preset codes and emerging themes.'* Data from the vari- 
ous sources were triangulated to determine whether each component, and subsequently 
the overall program, was implemented with weak or strong fidelity to the model. 


Academic Outcomes 


The primary academic outcome of interest was completion of the developmental math 
sequence. To provide context when interpreting ModMath’s effect on completing devel- 
opmental math, several additional academic outcomes are examined. These outcomes 
are described below. All academic outcomes were derived from the college’s administra- 
tive records, including transcript records, TSI placement test records, and MyMathTest 
placement scores. All 1,403 students in the analytic sample are included in analyses that 
examine the academic outcomes. Students without transcript records are considered not 
to have enrolled or to have made any further progress in math. All outcomes were 
measured at the end of the first, second, and third semester after random assignment. 
Enrollment: Two indicators of enrollment are considered: 


e Enrolled in college. Enrollment is defined as of the add-drop deadline. 
e Enrolled in any math class. Enrollment in any math class is defined based on 
enrollment in a module or a math course. 


Enrollment outcomes are important to consider for at least two reasons. First, it is 
possible that ModMath could increase persistence in college. This could occur if stu- 
dents find the approach more engaging than the traditional approach or if ModMath 
students perform better in math and consequently have a more positive school experi- 
ence. Second, our main math-focused outcomes are affected by enrollment (when stu- 
dents are no longer enrolled, they no longer make any progress in math), so enrollment 
rates provide important context when interpreting the math-focused outcomes. In terms 
of outcome levels, due to dropout alone we expect marginal increases in math progress 
to decline over time as larger proportions of students are not attempting any classes. 
This is also true with respect to impacts, where any program effects on enrollment have 
the potential to carryover to math progress. 

Developmental Math Progress/Completion: Several measures of developmental math 
progress and/or completion are examined: 


e Average percentage of developmental math sequence completed. One indicator 
of progress is the percentage of the developmental course sequence 


™~ codebook with preset codes was developed prior to field visits to classify data related to each core program 
component. Following each field visit, transcripts of interviews and focus groups were obtained using a professional 
transcription firm. These transcripts were uploaded to Dedoose, a software program for qualitative data analysis. 
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completed.'* Because students in ModMath can achieve smaller increments 
of success than students in traditional courses, this measure may favor the 
program group. 

e Earned at least one developmental math credit. Another indicator that a stu- 
dent made any progress is whether he or she passed at least one module (one 
credit) or one course (three credits) since random assignment. Since students 
in ModMath can complete a one-credit module and students in traditional 
courses must complete an all-or-nothing three-credit course, this measure of 
progress favors the program group. 

e Completed first half of developmental sequence. A final indicator of progress 
is whether students surpass the halfway milestone in the developmental 
sequence—Mod 3 or Math 0361 (the lower-level developmental math course). 
This may be the fairest progress milestone comparison since ModMath and 
the traditional course sequence have a similarly defined halfway point. 

e Completed developmental sequence (college-ready). The primary outcome of 
interest is completion of the developmental math sequence, which is defined 
as having passed Mod 6 or developmental Math 0362 since random assign- 
ment. Students achieving this milestone are considered “college-ready” in 
math and may proceed to college-level course work. 


Estimation 


To obtain a regression-adjusted estimate of the causal effect of the opportunity to par- 
ticipate in ModMath, we use the following general linear model: 


Y = yRB+ BT +e (1) 


where Y is the outcome of interest; RB is a vector of four random assignment block indica- 
tors (one for each unique cohort in the study); T is a treatment assignment indicator, set 
equal to one if a student is assigned to treatment and zero otherwise; and € is a random 
error term. Standard errors are Huber-White heteroskedastic robust standard errors.'* 
Estimated effects presented in the tables are the Gs for the relevant outcome. For simplicity, 
the main analyses presented in this article do not include any pretreatment control 


"For example, a program group student who completes Mod 1 but not Mod 2 is one-sixth of the way through the 
developmental sequence, or about 17%. A student completing Mod 2 but not Mod 3 has completed two-sixths of the 
sequence, or about 33%. For control group students, a student completing the developmental course Math 0361 (lower 
level math) has completed half of the sequence, or 50%, while a student completing developmental course Math 0362 
has completed the sequence. 

M4Clusters of students were in the same class together; however, we do not use cluster-robust standard errors as some 
researchers might. In this study, students were the unit of random assignment. As described by Abadie, Athey, Imbens, 
and Wooldridge (2017), “clustering is in essence a design problem, either a sampling design or an experimental design 
issue.” We did not sample clusters of students, and assignment to treatment was not clustered; thus, there is not 
support for making cluster adjustments due to sampling design or experimental design. 

In the absence of sampling clusters of students or cluster random assignment, cluster adjustment can still be 
justified if the estimand assumes the clusters in the study are like a random sample from a super population. Our 
target of inference is the effect of the opportunity to participate in ModMath for the students, classes, and instructors 
in the study, again implying no need for cluster adjustment. In addition, note that cluster robust standard errors tend 
to be larger than non-cluster robust standard errors. Since our primary findings are not statistically significant, using 
cluster robust standard errors should not change the statistical significance or substantive conclusions of this article. 
Finally, due to data limitations we cannot confidently determine which students were in which sections. 


APPROACH TO DEVELOPMENTAL MATH 497 


Table 5. First program semester math course type enrollment. 


Program group Control group Estimated Standard 

Outcome (%) (n = 826) (n= 577) effect error 
First semester 

ModMath 83.1 0.2 82.9*** 1.3 

Lecture-based 2.1 56.9 —54,.8*** 2.1 

Non-modularized computer-assisted 0.5 20.4 —19,9*** 1.7 

Other* 1.1 3.0 —1,9** 0.8 

No math course 13.3 19.6 —6.3*** 2.0 


Note. N= 1,403. Estimates are adjusted by cohort. Rounding may cause slight discrepancies in sums and differences. 
Calculations made using transcript data from Tarrant County College. 

“Includes math courses where the section type is unknown. 

ey = 01; **p = .05; *p = .10. 


variables. As a sensitivity check for the main outcomes of interest we re-ran the analyses 
including a set of covariates that could plausibly be related to outcomes. These analyses do 
not yield substantively different results than those presented. 

All analyses of academic outcomes are intent-to-treat and include all 1,403 students 
regardless of compliance with their experimental group. As discussed in more detail in 
the program implementation section below (see Table 5), 83.1% of the program group 
and less than 1% of the control group participated in at least some part of ModMath 
during the first semester after random assignment. Around 80% of the control group 
and less than 4% of the program group enrolled in another type of math course during 
that semester. The remaining students did not enroll in any math courses in the first 
semester after being randomly assigned. 

The probability of being assigned to the treatment group varied across random 
assignment blocks. To account for the varying treatment assignment probabilities, 
inverse probability of treatment weights are used to ensure an unbiased impact estima- 
tor.'° Weights were created as follows: 


ie =F 
on n(F) ac (Ta ) ms 


e 7, = 1 if individual i in random assignment block j was assigned to the pro- 
gram group and 0 if assigned to the control group 

e 7,;= the proportion of sample members in random assignment block j 
assigned to the program group (ie., the average value of 7 in random 
assignment block /) 

e 7..= the proportion of all sample members randomly assigned to the pro- 
gram group (i.e., the average value of Tj across all sample members) 


where: 


Results 
Program Implementation and Contrast Between Program and Control Conditions 
Overall, the program components were implemented with strong fidelity to the model 


and the experience of the program group was different in anticipated ways from the 


"Results are nearly identical with or without weights. 
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experience of the control group, as described in the program model section. 
Implementation fidelity was assessed based on adherence to the four program compo- 
nents: modular courses, diagnostic assessment, computer-assisted instruction, and on- 
demand, personalized assistance. 

Regarding modular courses, transcript records show that ModMath courses were 
offered as discrete 5-week, one-credit modules, as planned, and 83% of students in the 
program group participated in a ModMath course during their first semester post- 
random assignment. Qualitative data revealed that the ModMath and lecture-based 
course curricula were aligned, covering the same content. Regarding diagnostic assess- 
ment, placement test data indicate that more than 70% of students randomly assigned 
to ModMath took the MyMathTest diagnostic exam to determine their starting module. 
Regarding computer-assisted instruction, observational data indicate that ModMath 
classes were held in computer classrooms and instruction in ModMath classes was deliv- 
ered primarily via the MyMathLab software. Regarding on-demand, personalized assist- 
ance, observational data indicate that each class was staffed with an instructor and aide, 
who provided individualized assistance to students. Students would request help when 
needed, but instructors also proactively checked students’ work and offered assistance. 

In comparison, the control group enrolled in semester-long, three-credit courses. 
Transcript records show that there was variation in the types of courses control group 
students enrolled in. Approximately 57% of students enrolled in traditional lecture-based 
courses, and approximately 20% of students enrolled in one of the non-modularized 
alternative developmental math offerings, either computer-assisted lecture or Emporium. 
This is unsurprising as control group students could enroll in any developmental math 
course type, except ModMath. Table 5 shows the percentage of students who enrolled in 
each course type. (For simplicity, we have classified computer-assisted lecture and 
Emporium as non-modularized computer-assisted courses.) Regardless of the type of 
course in which they were enrolled, all control group students had access to the 
MyMathLab software package, as the software was required for homework. In addition, 
students in non-modularized computer-assisted courses used MyMathLab for in-class 
instruction to varying degrees. 

Even though students in both groups had similar access to the instructional software, 
which somewhat weakened the service contrast between the two groups, classroom 
observation and student survey data confirm a significant contrast between the class- 
room experience of ModMath program and control group students. During class, 
ModMath instructors and assistants focused on providing one-on-one assistance to stu- 
dents who were having difficulty with the course material. They responded to requests 
for help from students and checked in with students who did not request help, provid- 
ing unsolicited support. As shown in Table 6, 68% of ModMath survey respondents 
reported that their instructors spent a considerable amount, or most, of their time work- 
ing individually with students during class, compared with only 32% of control group 
students. In contrast, the majority of non-ModMath instructors focused class time on 
whole-group instruction. Seventy-nine percent of control group survey respondents 
reported that their instructors spent a considerable amount, or most, of their time lec- 
turing the class, compared with only 23% of ModMath students. 
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Table 6. Student survey results: Instruction and assistance. 


Sample Program Control Standard 

Response (%) size group group Difference error 
In your most recent math class, the instructor 

spent a considerable amount or most of 

the class period 

Lecturing the class 993 23.3 79.3 —56.0*** 2.7 

Working with small groups of students 992 22.8 25.0 —2.2 2.8 

Working with students individually 992 67.6 31.7 35,.9%** 3.1 

Giving announcements not related to math 989 12.4 14.4 -1.9 2.2 
In your most recent math class, the students 

spent a considerable amount or most of 

the class period 

Working alone on math exercises 992 80.5 53.6 26.8*** 3.0 

Working in small groups on math exercises 991 7.3 12.9 —5.6*** 2.0 

Working as a class on math exercises 990 15.1 60.5 —45,4*** 2.9 

Chatting, texting, or on personal business 991 41 6.3 —2.1 1.5 

Using computers, calculators, or technology 993 80.5 57.8 22.6*** 3.0 

Having problems with technology 993 25 5.9 —3.4** 1.3 


Note. Total survey respondents N= 1,012. Total program group respondents n= 620. Total control group respondents 
n=392. Estimates are adjusted by cohort. Rounding may cause slight discrepancies in sums and differences. 
Distributions may not add to 100% because categories are not mutually exclusive. Calculations made using data from 
survey of Tarrant County College students. 

***n = 01; **p = 05; *p = .10. 


Accordingly, there was a significant difference in how students spent their time in 
math class. ModMath students worked individually on the instructional software. 
Table 6 shows that approximately 80% of ModMath survey respondents reported spend- 
ing most of their time in class using computers, calculators, or technology and working 
alone on math exercises, while close to 60% of the control group reported similar expe- 
riences. This was expected given that the question asked whether students used com- 
puters, calculators, or technology and students in non-ModMath classes had time 
allotted for individual and small-group exercises. Generally, control group survey 
respondents reported more whole-group instruction. Sixty-one percent of the control 
group report spending a considerable amount, or most, of their time working as a class 
on math exercises, compared with only 15% of the program group. 

Taken together, these data highlight that ModMath offered students a more individu- 
alized class experience in which students spent significant time working independently 
on math exercises on their computers or being assisted by the instructor individually. In 
contrast, non-ModMath instructors spent a considerable amount of time lecturing the 
class, and students worked as a class on math exercises. 


Evidence for ModMath’s Theory of Change 


We examined whether the expected benefits of certain program components (as out- 
lined in the theory of change) were realized in practice. 

Fine-tuned assessment and module placement. Modules in conjunction with the fine- 
tuned diagnostic exam did not allow the majority of ModMath students to bypass 
already mastered material and start in a higher module. Approximately 84% of students 
placed at the beginning of the math course sequence under both testing approaches. 
Since these students likely needed remediation in all the content covered by the develop- 
mental math curriculum, the opportunity to place into modules using MyMathTest did 
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not make much of a difference.'® Relevant context when considering this finding is that 
when ModMath first began, prior to the present study, TCC offered three developmen- 
tal math courses and nine ModMath modules. Just prior to the start of the study, the 
lowest developmental math course was cut from the sequence due to a series of policy 
changes, and the number of modules was reduced from nine to six. This change may 
have resulted in a greater number of students beginning the developmental math 
sequence in the lowest course or module, limiting the potential benefit of the diagnos- 
tic assessment. 

Modularization and credit retention. As hypothesized (and shown later), the modular 
structure resulted in greater credit accumulation due to the ability to earn one credit at 
a time (see Program Effects on Academic Outcomes section below). Focus group data 
suggest that this resulted in a greater sense of accomplishment. For example, one par- 
ticilpant commented: “I’ve taken many, many remedial classes and haven’t gotten any- 
where because after a while I realized that I didn’t have the core that I needed to move 
on. I’m almost finished with my college career here at TCC, and the only thing that’s 
holding me back is the math requirements. So I was really upset that I had to start at 
the bottom [mod], but I will say I passed my first mod with an A, which I’ve never 
done that, so something must be working.” This illustrates a common sentiment by 
focus group participants who received transcript credit mid-semester for passing a mod- 
ule. By facilitating these small wins, ModMath is theorized to result in improved devel- 
opmental math completion. We examine whether this occurred in practice below. 

Content mastery. Due to the lack of a common final exam between program and con- 
trol group students, the current study could not directly assess whether ModMath resulted 
in improved content mastery. It is important to note that ModMath students could 
attempt their exams without working through the module content; however, they could 
not progress to the next module without passing the final exam for their current module. 

Self-paced learning and acceleration. Computer-delivered modules are theorized to 
improve student outcomes because they allow students to work at their own pace and 
potentially complete more than three modules in one semester. Our analyses revealed that 
acceleration was possible, but not prevalent. Focus group participants often referred to the 
self-paced nature of ModMath as being beneficial for the ability to slow down when learning 
the material, as opposed to accelerating and completing the sequence faster. For example, 
one student commented that with the video lessons, “you get to stop the teacher and keep 
going over and over [the material] until you get it, and you won’t move on, and you won't 
get left behind because the teacher or class is moving on.” This finding was confirmed by 
course transcript data, which revealed that more ModMath students repeated modules, as 
opposed to accelerating and completing more than three modules in one semester (24% ver- 
sus 1%, respectively). This occurred even though instructors provided students with a pacing 
calendar that detailed the amount of work they needed to accomplish each day to complete 
the module in the allotted time frame and the full course by the end of the semester. 

Although ModMath did not lead to student acceleration, the self-paced nature of the 
program may have resulted in students feeling that the difficulty of their course was 
appropriate. Approximately 71% of program group survey respondents reported that the 


'6myMathTest did appear to alter placement for 16% of students: Approximately 12% of students placed higher, and 
approximately 4% of students placed lower than under the college’s standard placement exams. 
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level of difficulty of their math class was just right, compared with 51% of control group 
survey respondents. 

Individualized instruction and student support. Based on interview and focus group 
data, ModMath increased one-on-one interactions between instructors and students, cre- 
ating increased opportunities for instructors to provide academic and emotional support 
to struggling students. ModMath focus group participants frequently juxtaposed their 
support experience in ModMath with their previous experience in lecture-based courses. 
For example, one student commented: “It’s so much easier than lectures because I don’t 
feel pressure to just hurry up and just understand it ... I don’t really like asking ques- 
tions in front of a big group of people because I’m scared I'm gonna ask a stupid ques- 
tion.” Most focus group participants felt that ModMath had increased their level of 
academic and personal support. 

In general, ModMath services were largely implemented as planned, and the program 
provided a notably different developmental math experience for students compared with 
TCC’s traditional lecture-based courses. There is evidence that some elements of theory 
of change were realized in practice, while others were not. The next section discusses 
the program’s impact on student academic outcomes. 


Program Effects on Academic Outcomes 


This section presents estimates of the effect of ModMath on students’ academic out- 
comes derived from the college’s transcript records and achieved over the course of 
three semesters after random assignment. Overall, ModMath was no more or less effect- 
ive than traditional math. After three semesters, 23% of program group members had 
completed the developmental math sequence, the primary outcome of interest. Similarly, 
22% of the control group reached the same milestone. On average, program group 
members completed a higher proportion of the six-credit developmental math sequence 
than did control group members. This was largely a result of the structural shift that 
allowed ModMath students to earn one credit for completing each of the six modules, 
whereas control group students earned credit only for completing the entire three-credit 
course. More details are provided below. 


Enrollment, Progress, and Completion of Developmental Math 


Derived from college transcript records, Table 7 presents information on students’ 
enrollment at college and progress through and completion of the developmental math 
sequence. Findings are presented for the first three semesters after students were ran- 
domly assigned; that is, regardless of cohort of entry, we present findings for each stu- 
dent’s first, second, and third semesters after entering the study.'’ Findings are 
presented semester by semester. 


“Recall that most program group members needed to pass all six modules and most control group members needed 
to pass two developmental math courses to complete the developmental math sequence; thus, three semesters 
provides a reasonable amount of time for students to achieve the goal of completing the developmental math 
sequence, although many students may take longer in practice. 
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Table 7. Developmental math progress. 


Program group Control group Estimated Standard 


Outcome (%) (N = 826) (N= 577) effect error 
First semester 
Enrolled in college 93.1 90.2 2.9* 1.5 
Enrolled in any math class (course or mod) 86.7 80.6 6:1°** 2.0 
Average percentage of developmental 
math sequence completed 26.4 17.5 8.9*** 1.5 


Developmental math sequence 
progress since RA 
Earned at least 1 developmental math 


credit or higher 69.5 30.0 39,5*** 2.5 
Completed first half of developmental sequence 27.1 29.8 —27 2.5 
Completed developmental sequence 
(college-ready) 2:2 5.1 —2,9*** 1.0 
Completed a college-level math course 0.4 0.3 0.0 0.3 
Second semester 
Enrolled in college 68.7 61.8 6.9*** 2.6 
Enrolled in any math class (course or mod) 51.4 40.3 11.2*** 2.7 
Average percentage of developmental 
math sequence completed 37.9 27.1 10.9*** 2.0 


Developmental math sequence 
progress since RA 
Earned at least 1 developmental math 


credit or higher 72.4 38.4 34,0*** 2.6 
Completed first half of developmental sequence 40.6 38.2 24 2.6 
Completed developmental sequence 
(college-ready) 14.7 15.7 =—1.0 1.9 
Completed a college-level math course 2.6 2.9 —03 0.9 
Third semester 
Enrolled in college 51.9 48.8 3.1 2.7 
Enrolled in any math class (course or mod) 35.3 29.9 5.5** 2.5 
Average percentage of developmental 
math sequence completed 43.2 32.5 10.7*** 2.1 


Developmental math sequence 
progress since RA 
Earned at least 1 developmental math 


credit or higher 74.4 43.2 31.2*** 2.6 

Completed first half of developmental sequence 45.7 42.8 2.9 2.7 

Completed developmental sequence 

(college-ready) 22.5 22.0 0.4 2.3 
Completed a college-level math course 6.9 9.2 —23 1.5 


Note. N= 1,403. RA=random assignment. Estimates are adjusted by cohort. Rounding may cause slight discrepancies in 
sums and differences. The Fall 2016 cohort has data only for part of the third semester. Calculations made using tran- 
script data from Tarrant County College. 

***n = 01; **p = 05; *p = .10. 


First semester. Immediately after random assignment, 93.1% of the program group 
and 90.2% of the control group enrolled at TCC. The difference, 2.9 percentage points, 
represents ModMath’s estimated effect on getting students to enroll in college.'* There 
are several possible explanations for this effect. This may reflect program group stu- 
dents’ preference for having been given the opportunity to participate in ModMath or 
control group students’ disappointment after not being offered the opportunity to par- 
ticipate in ModMath. Alternatively, it may be the case that ModMath staff were more 


"8Throughout this section the outcomes represent least squares means, and the estimated effect was calculated using a 
linear regression model. For ease of exposition we refer to the effect of ModMath, although technically it is the effect of 
the opportunity to participate in ModMath. Similarly, we refer to program and control group outcome levels, although we 
are presenting regression adjusted least squares means. 
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likely to conduct personal outreach to program group students to ensure they enrolled, 
than were non-ModMath staff for control group students. 

In addition to this small positive effect on enrolling in college, students offered 
ModMath were 6.1 percentage points more likely to enroll in a math class (a module or 
a course) than were their control group counterparts.’ In other words, assignment to 
ModMath caused an estimated 50 additional students to enroll in math (out of the 826 
program group students).”° This positive effect on attempting any math credits is intri- 
guing, since, as noted by Bailey et al. (2010), “More students exit their developmental 
sequences because they did not enroll in the first or a subsequent course than because 
they failed or withdrew from a course in which they were enrolled.” 

Notably, to be eligible for the study, students had to be willing to participate in 
ModMath, and the only way to get into ModMath was to enroll in the study. Students 
who preferred the traditional math course could simply enroll in it, without participat- 
ing in the study. Thus, ModMath’s positive effect on getting students to take a math 
class may apply only to the types of students in the evaluation and may not generalize 
to all students in the college who required developmental math, especially those who 
preferred the traditional course. Nonetheless, the offer of ModMath encouraged some 
additional students to at least attempt a developmental math class, which is a strong 
starting point." However, the goal of ModMath is to help students progress through 
and ultimately complete the developmental math sequence. We turn to progress and 
completion next. 

Program group students were much more likely to earn at least one developmental 
math credit (by passing a module or a course) than were their control group counter- 
parts—an indication of at least some degree of progress. Nearly 70% of program group 
students completed at least one class (typically at least one 5-week, one-credit module), 
compared with only 30% of control group students (typically a semester-long, three- 
credit course). This 39.5 percentage point increase in earning at least one math credit 
occurred largely because ModMath students had the opportunity to pass smaller por- 
tions of the developmental math sequence, 5 weeks at a time, one credit at a time. 
In contrast, students enrolled in the traditional courses were in an all-or-nothing 
situation—they had to pass an entire 16-week course to earn three credits. This struc- 
tural difference led to a substantial difference in the number of students who were able 
to make some formal progress in the developmental sequence by accumulating one or 
more developmental math credits during the first semester. 

Relatedly, on average, program group students completed a higher percentage of the 
developmental math sequence than did control group students. By the end of the first 
semester, program group students had completed 26.4% of the developmental math 
sequence, whereas control group students had completed 17.5% of the sequence—for an 
estimated effect of 8.9 percentage points. The positive effect on the percentage of the 
developmental math sequence completed is also due, in large part, to program students 


Note that 0.6% of program group members and 0.3% of control group members enrolled directly in a college-level 
math course in their first semester after random assignment. 

°Calculated as 826 x 0.061 = 50. 

"Like the small effect on initial enrollment in college, the reason for the positive effect on attempting a developmental 
math course is not certain. It may have been due to the appeal of the program, but it also may have been an artifact 
of the experiment. 
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earning credit for completing one or two modules in a semester, rather than having to 
complete an entire three-credit course.” 

ModMath’s positive effect on completing at least one math credit and on the percent- 
age of the developmental math sequence completed is encouraging. However, this suc- 
cess is tempered by the fact that only 27.1% of the program group completed the first 
half of the developmental sequence (Mod 3) by the end of the first semester. Students 
in the traditional math sequence had a similar likelihood of success: 29.8% of the con- 
trol group made it through the halfway point (Math 0361) in one semester. The negative 
2.7-percentage point estimated effect is not statistically significant, but it is discourag- 
ing. Moreover, there is evidence that ModMath lowered the proportion of students who 
completed the developmental sequence (and were thus college-ready) during their first 
semester. Only 2.2% of program group students accomplished this goal, compared with 
5.1% of control group students, for an estimated effect of negative 2.9 percentage points, 
representing around 24 people. 

In sum, after one semester the effects of ModMath were mixed. The program helped 
students make greater progress on average, but program group students were no more 
likely than control group students to reach the halfway milestone, and they were slightly 
less likely to complete the full developmental math sequence. Does the “early win” of 
earning at least one math credit, achieved by nearly 70% of the program group, translate 
into larger gains in future semesters? We turn to this next. 

Second semester. ModMath increased second-semester enrollment by an estimated 6.9 
percentage points. The program group’s second semester enrollment rate was a disap- 
pointing 68.7 percent, implying a high level of dropout (or stop out); however, this rate is 
encouragingly significantly higher than the control group’s enrollment rate. Much like the 
first semester enrollment effect, this finding may have to do with program group students’ 
positive experiences in ModMath or the control group’s disappointment of not being 
offered ModMath. In this case, the effect estimate is positive and highly statistically sig- 
nificant (p = .008), so it is unlikely a chance finding. Notably, part of this effect is driven 
by the 7.2% of program group members who only enrolled in a math class (compared 
with 3.1% of the control group). This may suggest that the overall enrollment effect is 
partially driven by structural differences, like the fact that the census date for the first 
Mod was one week into the semester and the census date for regular courses was two 
weeks into the semester. However, among those program group students whose only class 
was math (n= 60), 70% earned one or more credits and 43% earned two or more credits. 
That is, most program group students who enrolled only in math did accomplish some- 
thing; they did not simply enroll and then drop out after the census date. So, the overall 
enrollment effect appears to be a bright spot in an otherwise sobering story. 

Regarding math progress and completion, during the second semester program and 
control group students continued to make progress. On the positive side, ModMath 
again caused more students (11 percentage points) to enroll in a math class; students 
offered ModMath remained much more likely (34 percentage points) to have earned at 
least one math credit since the start of the study; and the program group maintained its 
advantage with respect to the percentage of the developmental course sequence 


Recall that control group members can only be 0%, 50%, or 100% complete, whereas program group members can 
be 0%, 33%, 50%, 67%, 83%, or 100% complete. 
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completed (10.9 percentage points). However, ModMath had no discernable effect on 
causing students to achieve the key milestones of completing the first half of the devel- 
opmental math sequence or completing the entire sequence and thus being deemed col- 
lege-ready in math. Lamentably, only around 15% of the 1,403 students in the study 
became college-ready in math within two semesters. Most students either did not 
attempt all required developmental math courses/modules, failed one or more courses/ 
modules, or did not reenroll at TCC. 

Third semester. Students were tracked through three semesters after they entered the 
study. Students in ModMath and the more traditional math course had almost identical 
rates of completing the developmental math course sequence (22.5% vs. 22.0%, respect- 
ively). This occurred even though program group students continued to attempt a math 
class (course or module) at a higher rate than their control group counterparts, and 
they maintained their advantage with respect to the percentage of the developmental 
math sequence completed. Nonetheless, when it came to three major milestones—com- 
pleting the first half of the developmental math sequence, becoming college-ready in 
math, and passing the first college-level math course—there was no discernable differ- 
ence between the outcomes of students offered ModMath and their control group 
counterparts. 


Non-Math Progress. Table 8 examines non-math credits attempted and earned through 
three semesters. We find no evidence of any discernable positive spillover effects or 
negative unintended consequences of ModMath. 


Subgroup Findings 


In addition to examining ModMath’s overall average effects, we explored the programs’ 
effects for different types of students—specifically, with respect to students’ baseline 
comfort with technology, developmental need, intent to enroll full time, and intent to 
work full time—all of which were measured prior to random assignment. Table 9 
presents findings by subgroup. For example, the first panel shows that ModMath’s esti- 
mated effect on becoming college-ready in math after three semesters is 1.0 percentage 
point for students who self-reported being comfortable with technology at the start of 


Table 8. Cumulative non-math progress. 


Program group Control group Estimated Standard 

Outcome (n = 826) (n=577) effect error 
One semester 

Non-math credits attempted 6.49 6.56 —0.07 0.22 

Non-math credits earned 4.82 4.73 0.10 0.23 
Two semesters 

Non-math credits attempted 11.76 11.67 0.10 0.43 

Non-math credits earned 8.64 8.49 0.14 0.42 
Three semesters 

Non-math credits attempted 15.49 15.16 0.33 0.61 

Non-math credits earned 11.44 11.14 0.30 0.58 


Note. N= 1,403. Estimates are adjusted by cohort. Rounding may cause slight discrepancies in sums and differences. 
The Fall 2016 cohort has data only for part of the third semester. Calculations made using transcript data from Tarrant 
County College. 

***n = 01; **p = .05; *p = .10. 
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the study and 0.1 percentage points for students who were not. The final column in the 
table, with p value 0.867, shows that these two effect estimates are not statistically distin- 
guishable. Stated differently, the 0.9 percentage point difference (1.0 — 0.1) in effect esti- 
mates between these two groups could easily have occurred by chance if the program’s 
true effects were the same for both groups. The rest of the table shows that there is not 
clear evidence that ModMath was more effective for any subgroups of students: Effect 
estimates are mostly near zero, just like the overall average. 

In summary, breaking the developmental math course sequence into six one-credit, 
computerized modules led ModMath students to make incremental progress toward 
completing the sequence. Despite this apparent advantage, the ModMath program is no 
more (or less) effective than the traditional developmental math course at helping stu- 
dents complete their developmental math requirements. 


Limitations 


This study is one, if not the only, experimental evaluation of a modularized, computer- 
assisted, self-paced developmental math program in a community college setting. As a 
result, it provides an internally valid estimate of the causal effect of such a program on 
students’ completion of the developmental math course sequence. 

One important limitation of the study is that we are unable to assess ModMath’s 
effect on mastering course content since program and control students did not take a 
common standardized exam or posttest and it was beyond the scope of the project to 
ensure that they did so. This is a common challenge in postsecondary research, where 
truly standardized outcomes are rare. That said, upon completing the developmental 
math course sequence, program and control group students could enroll in the same 
college-level math courses, at which point any deficit or benefit with respect to content 
knowledge could result in higher pass rates; we do not observe any program effects on 
passing college-level math courses.” 

Another limitation is that the present study’s findings are specific to ModMath at TCC 
and may not generalize to other colleges with different institutional contexts, different 
alternative course offerings, or drastically different student populations. Moreover, stu- 
dents could enroll in ModMath only if they joined this study, while students could enroll 
in other course types without restriction. As a result, the findings presented here do not 
necessarily inform whether ModMath would be well suited for students with a strong 
preference for traditionally structured lecture classes; such students could experience 
ModMath differently and therefore have different academic outcomes. 

Finally, we did not conduct a formal cost analysis as part of this project. Nonetheless, 
with respect to cost, it is worth noting that ModMath likely costs a little more than the 
traditional math class. Although ModMath and traditional math classes were of similar 
class size, and both offered the use of instructional software, only ModMath included an 


?3The TCC math department staff, who developed ModMath and operate the other math course options, consider 
completion of each of the various alternative course sequences to be equivalent. A standard curriculum was used for all 
the college’s developmental math course sequence options (including ModMath), and completing any option enables 
students to meet an institutional requirement. Moreover, since the overall conclusion is a story of null effects, the 
standards across treatment and control groups would have to be large to substantially change the findings. Given that 
the intention of the math department is to keep the standards the same, this may not be likely. 
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instructor and an instructor’s aide. The additional cost per student for an aide is an 
important consideration when interpreting the findings from this study. 


Discussion 


Increasing the academic success of students referred to developmental math courses is a 
pressing priority for community colleges. Modularized, computer-assisted, self-paced remed- 
ial math courses, such as ModMath, are a widespread reform currently being implemented 
at institutions across the nation, yet limited rigorous evidence on the efficacy of this 
approach exists. The present study employed a randomized controlled trial to evaluate 
ModMath and found that ModMath is similarly effective at getting students through the 
developmental math sequence as other course formats, particularly traditional lecture- 
based courses. 

Sometimes when an evaluation finds that an intervention was no more or less effect- 
ive than the alternative, the lack of impact can be attributed to poor program imple- 
mentation or a weak contrast between the program and control groups. This was not 
the case in this study: For the most part, the program services were implemented as 
planned by its designers and there were significant differences between the classroom 
experiences of ModMath and control group students. Given this, we look to the theory 
of change for insight into the findings. 

One potential explanation for ModMath’s lack of positive impact is that some elements 
of the theory of change that were theorized to lead to improved student outcomes were 
not realized in practice. The fine-tuned diagnostic exam did not change placement for the 
majority of ModMath students; the self-paced modules did not result in student acceler- 
ation; and the small wins gained by earning credits incrementally did not increase com- 
pletion of the developmental math sequence. We discuss each of these, in turn. 

With regard to the diagnostic exam, it is possible that the curricular changes that 
shortened the developmental math sequence from three to two courses just prior to the 
start of the study muted the potential benefits of the fine-tuned assessment. As a result, 
the college may consider alternative placement methods. Many institutions are begin- 
ning to rely on multiple measures to assess student readiness for college-level work, 
instead of relying on a single exam, and recent studies have shown that this reform may 
improve students’ outcomes in math (Barnett et al., 2018; Scott-Clayton & Belfield, 
2015). For TCC, incorporating multiple measures of assessment (in addition to a place- 
ment exam) may better identify students who would succeed in a higher developmental 
math module or course. 

With regard to self-pacing, we find that more ModMath students slowed down as 
opposed to accelerated, which is consistent with other research on self-paced computer- 
assisted courses and may be relevant beyond developmental math education. There is a 
tension among autonomy, mastery, and acceleration in self-paced, modularized courses 
(Bickerstaff et al., 2016). Mastery learning requires students to meet certain benchmarks 
before moving forward in the curriculum, which will naturally lead to slower progres- 
sion. Furthermore, since many developmental math students have a history of underper- 
formance in math, and many community college students manage school with work 
and other personal responsibilities, such as childcare, the flexibility offered by a self- 
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paced course makes it more likely that these students will progress more slowly as 
opposed to more quickly. While ModMath designers attempted to overcome this chal- 
lenge by issuing pacing calendars to keep students on track, acceleration was not expli- 
citly promoted. To promote acceleration in ModMath, and other self-paced courses, 
while addressing the academic and personal needs of developmental math students in 
community colleges, a triage approach may be necessary. Students who are more 
advanced or have more available time could be identified and explicitly encouraged to 
move faster, through personalized pacing calendars or other means. 

Alternatively, developmental math acceleration strategies that tailor and align required 
content to student’s academic interests, as opposed to expecting students to cover more 
material in less time, may be more feasible. These types of strategies are gaining in popu- 
larity and some are showing early signs of success. For example, a study on the Dana 
Center’s Math Pathways program, which reduces the required number of developmental 
math courses based on program of study, has shown early signs of positive impacts on 
college readiness (Zachry Rutschow, 2018). Similarly, a randomized controlled trial of an 
initiative that shifted required course content, from a prerequisite elementary (remedial) 
algebra course to statistics with co-requisite remediation that only addresses the math 
needed for the statistics course, for students in non-STEM (science, technology, engineer- 
ing, and mathematics) majors is gaining in popularity, and at least one randomized trial 
finds evidence of improved outcomes (Logue, Watanabe-Rose, & Douglas, 2016). 

Regarding modularization, we find that ModMath’s modular course structure resulted 
in greater credit accumulation and increased feelings of accomplishment for students, 
but these small wins were not enough to get students to the finish line. The fact that 
only about 22% of students in either group completed the two-semester developmental 
math sequence in three semesters indicates that far too many students are struggling 
using either approach. 

Given this, the fact that higher education institutions are searching for ways to get 
students through the developmental math sequence faster, and early experimental 
research showing more positive results for other types of interventions (Barnett et al., 
2018, Logue et al., 2016; Scrivener et al., 2018; Zachry Rutschow, 2018), ModMath is 
not currently the most promising strategy for developmental math reform. It is, how- 
ever, an alternative to traditional lecture-based courses that leads to similar outcomes. 
Thus, to accommodate individual student preferences, colleges could offer ModMath 
and expect similar results as lecture-based courses. 
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